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ABSTRACT 



This document focuses on assessment portfolios and examines 
the advantages and challenges of using an assessment portfolio system 
inclusive of English language learners as a district-wide assessment tool. 
Advantages of including English language learners in the assessment program 
include increased school accountability for all students, a shared vision of 
student goals and standards, and an authentic picture of student learning. 
Assessment portfolios that include English language learners really can 
reflect assessment reforms. Among the drawbacks to the use of portfolio 
assessments that include English language learners is the fact that 
reliability is lower than for more traditional assessments, making 
comparisons more difficult. It is difficult to ensure standardized testing 
conditions, and the cost of administering portfolio assessments is higher 
than that of many other forms of assessment. Portfolios also pose a scoring 
dilemma because of staff development and training requirements and the item 
required to score portfolios correctly. Developing and implementing a 
large-scale assessment portfolio that includes English language learners 
requires extensive planning and discussion and requires considerable 
resources. Some steps to developing and implementing such a program are 
outlined. (Contains 16 references.) (SLD) 
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Creating Large-Scale Assessment Portfolios 
That Include English Language Learners 



Are Assessment Portfolios That Include English 
Language Learners Right for Your State or District 1 

■ Are you looking for an assessment system that includes English language 
learners? 



■ Do you want to ensure that schools are accountable for ALL students, 
including students learning English? 

■ Do you want practitioners to reach a shared vision of student goals, aligned 
to content standards? 

■ Are you looking for an assessment system that provides an authentic 
picture of student learning? 

■ Do you want to promote improved teaching and student learning? 

■ Do you want your assessment system to reflect assessment reform? 




U.S DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

B^rnis document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



BEST COPY AVAILABLE 

1 



2 








- ^ 



rm 




March 1999 



QJ> 

*4— * 

u 

cs 

J-H 

T3 

C 

>- 

'o 

C 

o 



CO 

W 




H 

u 

W 

C^H 

CO 

w 





Creating Large-Scale Assessment Portfolios 
That Include English Language Learners 



Are Assessment Portfolios That Include English 
Language Learners Right for Your State or District? 

■ Are you looking for an assessment system that includes English language 
learners? 



■ Do you want to ensure that schools are accountable for ALL students, 
including students learning English? 

■ Do you want practitioners to reach a shared vision of student goals, aligned 
to content standards? 

■ Are you looking for an assessment system that provides an authentic 
picture of student learning? 

■ Do you want to promote improved teaching and student learning? 

■ Do you want your assessment system to reflect assessment reform? 




Introduction 



In f V he current school reform movement has emphasized that the education of all 
students must be improved. Assessing the achievement of every student is an 
essential part of school reform, but it is a challenge for most schools, districts, 
and states (North Central Regional Educational Laboratory & Council of Chief State 
School Officers, 1996). Traditional assessment practices of many states and districts have 
excluded a population of students known as English language learners or limited English 
proficient students. Consequently, English language learners are often denied access to 
important educational opportunities that are based upon assessment results. 

This document focuses on one type of assessment system, assessment portfolios, and 
examines the advantages and challenges of using an assessment portfolio system 
inclusive of English language learners as a district-wide assessment tool. The audience 
for this document is district-level administrators who are responsible for establishing 
district-wide assessment programs, as well as state education agency staff who provide 
technical assistance to districts. 



a 



What is an 
assessment 
portfolio? 

A n assessment portfolio is the 

systematic collection of student work 
measured against predetermined scoring 
criteria. These criteria often take the form 
of scoring guides, rubrics, check lists, or 
rating scales (O’Malley & Valdez Pierce, 
1996). Portfolios are like a slide show that 
demonstrates student achievement, rather 
than the snapshot of student achievement 
that single-occasion tests provide. Assess- 
ment portfolios can include performance- 
based assessments, such as writing sam- 
ples that illustrate different genres, solu- 
tions to math problems that show 
problem-solving ability, lab reports 
demonstrating an understanding of a 
scientific approach, or social studies 
research reports demonstrating the ability 
to use multiple sources. In addition, 
district-wide assessment portfolios can 
include scores on commercially- 
developed, nationally norm-referenced 
tests, such as the Iowa Test of Basic Skills 
(ITBS) or the California Test of Basic Skills 
(CTBS), or results of state assessment 
measures such as the Maryland School 
Performance Assessment Program 
(MSPAP), and other information 
pertaining to students’ educational 
background. □ 



Portfolios are like a slide 
show that demonstrates 
student achievement , 
rather j than the snapshot 
of student achievement 
that single-occasion tests 
provide . 




What are the advantages of assessment 
portfolio systems for English language learners? 



Inclusion of English 
language learners 

U ' nlike other assessment programs, 

1 assessment portfolios do not exclude 
certain student populations from participa- 
tion. A major problem with commercially- 
developed, norm-referenced and criterion- 
referenced testing programs is that English 
language learners are often excluded from 
these tests because of their lack of profi- 
ciency in English. Many states have policies 
that restrict English language learners from 
taking these exams for one to three years 
after their arrival in a U.S. school system, or 
until a certain level of proficiency is reached 
(Special Issues Analysis Center, 1995). 

These same tests act as gatekeepers to some 
educational opportunities, such as programs 
for gifted and talented students, magnet 
school participation, and even Title I 
services. As a result, exclusion from the 
system-wide testing program deprives 
English language learners from access to 
important educational opportunities that are 
based upon the results. In contrast, 
assessment portfolio systems include 
samples of student work that are collected 
and scored according to predetermined 
criteria. Therefore, all students, including 
English language learners, can be included 
in the assessment system. 

English language learners are included 
in the assessment programs of two states 
(Vermont and Kentucky) that use assess- 
ment portfolios. In Vermont, the Portfolio- 
Based Writing Assessment Program was 
developed to measure how well Vermont 
schools teach students to write across the 
curriculum, and to “provide a high common 
standard of achievement for all students” 
(Koretz, Stecher, & Diebert, 1992, p. 2). 
Similarly, the Kentucky writing and mathe- 
matics portfolios are integral parts of the 
Kentucky Instruction Results Information 
System (KIRIS). The writing and math port- 
folios are collections of students’ best work, 
selected from particular classroom activities, 
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which Kentucky teachers score using scoring 
guides for writing and mathematics along 
with benchmarks of student work (Rigney, 

S., personal communication, January 3, 
1997). Admittedly, the number of English 
language learners in these states is small, 
and information pertaining to these students 
is not disaggregated. However, in both of 
these assessment portfolio programs, all 
students in the schools that participate, 
including English language learners, are 
assessed. Thus, the schools are held ac- 
countable for every student who is assessed. 

Increased school accountability 
for all students 

At the state and district level, assessment 
information is often collected to ensure that 
the educational system addresses the needs 
of all of its learners. Inclusion of English 
language learners in state or district-wide 
testing programs is critical for accountability 
and to produce accurate data reflecting their 
achievement (O’Malley & Valdez Pierce, 
1994). By providing systems with a richer 
source of information about student learn- 
ing, assessment portfolios can help school 
systems identify and meet the needs of 
diverse students, including English language 
learners. 

Large-scale assessment portfolio pro- 
grams assess specific abilities or achieve- 
ment levels according to criteria that have 
been agreed upon by teachers, administra- 
tors, and other stakeholders within the 
system. Thus, expectations are clearly 
defined and can be disseminated to all 
stakeholders, including students, teachers, 
and parents. The results are used to 
evaluate school and district services. 

Assessment portfolios can increase 
accountability for the education of English 
language learners because portfolios contain 
at least some information upon which to 
base educational decisions, even if one item 
of the portfolio is missing from an English 
language learner’s file (e.g., the score from a 
norm-referenced test). If the results show 



that English language learners as a group 
are not doing well, this information can be 
used over time to guide districts and states 
to align resources to improve English 
language learners' achievement. 

Because traditional testing policies that 
utilize commercially-developed, norm- 
referenced or criterion-referenced tests often 
exempt English language learners from 
taking these tests during their initial years in 
U.S. educational systems, many schools 
have not been held accountable for the 
achievement of a large majority of English 
language learners in these important years 
(O'Malley & Valdez Pierce, 1994). 

A shared vision of student 
goals and standards 

Developing an assessment portfolio system 
that includes English language learners 
helps teachers, administrators, parents, and 
students shape a common vision of what 
students should know and be able to do as 
a result of their course work. The process 
also requires that English as a second lan- 
guage (ESL) and bilingual program teachers 
participate in the discussion, improving the 
chances that the assessment tasks will 
reflect important issues in second language 
acquisition. For example, when assessing 
content knowledge, teachers must look 
beyond language errors and focus on how 
well students demonstrate an understand- 
ing of the content concepts. By clearly 
articulating expectations and the criteria 
upon which to assess attainment of these 
expectations, school systems help create a 
shared vision of the purpose of education 
based on the values of the community. 

In addition, when stakeholders work 
together to develop shared goals, they can 
address the needs of English language 
learners. Language and content standards 
can be represented in both ESL and content 
area curricula. These goals may be directly 
linked to state content and performance 
standards. Drawing from this vision, school 
systems can identify, select, and/or create 
assessments that represent the educational 
goals they hold for their students. Scoring 
O i can then be established, and 
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benchmarks of varying achievement levels 
can be provided to collect information on 
student attainment of the standards. As a 
result of the standards-based reform 
movement, a number of states and districts 
have revised their curricula. As yet, few 
districts have used content standards 
appropriately for English language learners 
and measured attainment of the standards 
with valid and reliable assessments. In San 
Diego, California, the Literacy Assessment 
Portfolio piloted in the 1996-97 school year 
measured the progress of students in a 
program that has aligned San Diego's 
English language development program to 
the district's English language arts stan- 
dards. The reading portion of the assess- 
ment portfolio system was fully imple- 
mented in the 1997-98 school year and 
assesses all students district-wide in grades 
K-12, including those who speak a first 
language other than English. Portfolios at 
grades 3, 4, and 8 are scored centrally, with 
scores on the reading exhibit reported as 
part of the district's accountability plan. 

San Diego is one of the first districts in 
the United States to assess all students 
using an assessment portfolio system that 
includes English language learners and is 
premised upon a shared vision of standards- 
based education (Burns, G., personal com- 
munication, May 12, 1997). When used as 
part of a multi-phased assessment program, 
assessment portfolios can also be used in 
the formative evaluation process as state 
<and district staff implement standards-based 
curricular reforms. 

An authentic picture of 
student learning 



Developing an assessment 
portfo io system that includes 
English language learners 
helps teachers, administrators, 
parents, and students shape a 
common vision of what 
students should know and be 
able to do as a result of their 
course work . 
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stration of student achievement. 

Assessment portfolios are one of the few 
assessment procedures that accommodate 
learners' specific needs (Ancess & Darling- 
Hammond, 1994). Portfolios are designed to 
be inclusive and contain an authentic 
description of what students can do. Even 
English language learners at the beginning 
level of English proficiency can be asked to 
write a story and then judged according to 
scoring criteria. Likewise, students with 
limited English proficiency can perform 
math or science activities. 

Proper accommodations, such as ex- 
tending time or allowing the use of diction- 
aries, can create an environment in which 
English language learners maximize their 
opportunity to demonstrate knowledge. 
Because assessment portfolios usually con- 
tain complete products, educators recognize 
that portfolios are valid representations of 
student learning. For this reason, several 
school districts across the country have 
implemented district-wide assessment port- 
folios for their English language learners, 
including San Diego; Arlington County, 
Fairfax County, and Prince William County 
in Virginia. 



Assessment portfolios can be designed to 
measure virtually any observable skill or 
process, or content-area knowledge, needed 
for system-wide assessment purposes. In 
fact, a wide range of student products can 
be included in assessment portfolios and 
used to assess student learning and achieve- 
ment, as long as predetermined scoring 
criteria are in place (O'Malley & Valdez 
Pierce, 1996). As noted earlier, portfolios 
provide educators with an ongoing demon- 



improved teaching and 
student learning 

Using assessment portfolios that include 
English language learners not only provides 
improved information about student 
achievement, but also makes a positive 
impact on teaching and student learning. In 
Vermont schools that have elected to 
participate in the statewide writing 
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When teachers are trained to 
use and score portfolios based 
on agreed-upon criteria, they 
tend to move toward a more 
learner-centered teaching 
modell which encourages stu- 
dents to take more responsi- 
bility for their own learning . 



assessment (92% of all public schools), all 
teachers at the 5th and 8th grade levels have 
been involved in staff development on the 
use of writing portfolios, and many have 
been trained intensively to improve inter- 
rater reliability among scorers. 

While the training has been a major staff 
development effort, Geof Hewitt, a writing 
assessment consultant for the state, noted 
the program not only provides state officials 
with a richer understanding of how well 
students write, but has the added effect of 
improving teaching and learning based on 
the feedback that portfolios provide to 
teachers and students (Hewitt, G., personal 
communication, October 11, 1996). In fact, 
after studying Vermont’s portfolio assess- 
ment program, researchers from the RAND 
Corporation concluded that the effects of 
assessment portfolios on instruction were 
“substantial and positive” (Koretz, Stecher, 

& Diebert, 1992). 

Likewise, an evaluation of Kentucky's 
portfolio assessment program conducted by 
Western Michigan University found that 
students in Kentucky were writing more and 
doing more group work as a result of the 
new state testing program. Teachers and 
other stakeholders reported “almost unani- 
mously” that student writing had improved 
in Kentucky (Baker, 1996). 

When teachers are trained to use and 
score portfolios based on agreed-upon 
criteria, they tend to move toward a more 
learner-centered teaching model, which 
encourages students to take more respon- 
sibility for their own learning. Through 
training, teachers develop an understanding 
of the quality of student work that meets 
snprific achievement levels, according to the 



scoring criteria. During training sessions, 
teachers go through the process of judging 
portfolio samples and scoring them accord- 
ing to the criteria defined in the scoring 
rubric. While doing this, they often clarify 
what objectives the instructional program is 
designed to meet. In turn, teachers often 
modify their own teaching to provide stu- 
dents with a better understanding of those 
same instructional objectives. The process of 
developing, implementing, and scoring 
portfolios takes a concerted effort, and 
requires a truly precious commodity - time. 
Yet the training effort results in a more 
learner-centered classroom. 

Thus, while scoring portfolios is a time- 
intensive task, it is time well-spent. Assess- 
ment portfolios provide states and districts 
with a better understanding of student 
ability and achievement and allow both 
students and teachers to evaluate work with 
a common vision of excellence. The teacher 
training that is necessary to implement 
assessment portfolios also prepares teachers 
to teach to an increasingly diverse student 
population, and, more importantly, leads to 
improved student achievement. 

It is critical, however, to provide quality 
staff development programs that include a 
discussion of second language acquisition 
and its impact on learning, so school sys- 
tems can implement an assessment portfolio 
program with all students, including English 
language learners. Improved teaching 
results from such staff development that 
prepares teachers to use and score portfolios 
equitably. 

Assessment portfolios reflect 
assessment reform 

Advocates of assessment reform call for new 
measures that provide a better understand- 
ing of student achievement, especially for 
English language learners. In a report 
entitled Quality Counts: A Report on the 
Condition of Public Education in the 50 
States , Education Week and the Pew 
Charitable Trusts define crucial data that 
states do not currently collect that would 
help states assess their educational systems 
(Wolk, 1997). Some of the uncollected data 



include: whether states' standards set 
rigorous expectations; whether states' 
assessments adequately measure student 
achievement of the standards; whether 
states' exit exams are linked to core content 
standards; whether states have explicit 
performance standards that include samples 
of student work; the percentage of the 
student population excluded from the 
assessments; and the percentage of teachers 
participating in professional development 
grounded in new content standards. By 
using assessment portfolios inclusive of 
English language learners, states would 
acquire information related to several of 
these data points. 

When educators think about implement- 
ing assessment portfolios, they often initiate 
a concerted discussion about achievement 
expectations in a state or district. Assess- 
ment portfolios that are the result of such 
discussions can be directly connected to 
content standards in the system. By imple- 
menting assessment portfolios that include 
English language learners, school systems 
could reduce the number of students ex- 
cluded from the system-wide assessment 
program and possibly increase the number 
of teachers participating in professional 
development activities. 

Finally, assessment portfolios can also be 
administered in a similar way to the NAEP 
(National Assessment of Educational Pro- 
gress), an integral piece of assessment 
reform. The NAEP, considered to be one of 
the best-designed assessments used nation- 
ally, has recently begun to make accommo- 
dations to include more English language 
learners in their national sample. Accommo- 
dations for students who speak a language 
other than English, such as translated 
directions, dual language or translated test 
booklets, extended time periods, and small 
group administration, have all been recom- 
mended or are under consideration by NAEP 
(Olsen, 1996). Similar modifications can be 
employed in district-wide assessment port- 
folio systems that include English language 
learners, either to the text of portfolio 
entries, or to the testing procedures (Butler & 
Stevens, 1997). □ 



What are the 
challenges of 
assessment portfolios 
that include English 
language learners? 

Lower reliability and comparability 

A ssessment portfolios are not without 
challenges as a system-wide assess- 
ment tool. First, state education agencies 
report that there is a general public resis- 
tance to more performance-based testing 
because it is perceived to diminish the ac- 
countability of schools and districts (Special 
Issues Analysis Center, 1995). The public 
has become accustomed to single scores, 
such as those used to describe the results of 
standardized, norm-referenced tests, like the 
CTBS. Single scores are comparable across 
systems, and from one year to the next. 
However, many performance-based tests, 
including some portfolio systems, do not 
easily or meaningfully translate into a single 
score, or set of scores, that can be readily 
compared. Because some districts report the 
results of performance-based tests in words, 
rather than with a numerical score, stake- 
holders sometimes feel the school system is 
less accountable for individual students. 

Furthermore, it is difficult to implement 
assessment portfolios that meet the reliabil- 
ity requirements many school systems want. 
Achieving a certain degree of reliability 
among raters or test evaluators (for example, 
.7 agreement or better) is important, accord- 
ing to researchers at the National Center for 
Research on Evaluation, Standards, and 
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Contrary to public 
perceptions, however, 
standardized testing 
conditions can be 
implemented for specific 
contents of assessment 
portfolios, such as a writing 
prompt . 



Student Testing (Novak, Herman, & Gear- 
hart, 1996). Without achieving high inter- 
rater agreement figures, the usefulness of 
the scores as an accountability tool diminish 
. because the results cannot be used to 
compare scores reliably between schools 
within the system, or over time. 

However, assessment portfolios can be 
designed with the single score requirement 
in mind. For example, Vermont’s inter-rater 
agreement figures on the writing portfolios 
actually improved when they averaged 
scores on five writing subsections into a 
single score (Koretz, McCaffrey, Klein, Bell, 

& Stecher, 1993). On the other hand, if the 
results of assessment portfolios are trans- 
lated into numerical scores, the benefit of 
richer information about student learning is 
lost. Therefore, it is recommended that both 
results be retained. 

Difficulty ensuring standardized 
testing conditions 

Some states and districts are prevented from 
implementing performance-based assess- 
ment programs that include assessment 
portfolios by state legislatures mandating 
the use of traditional norm-referenced tests. 
One reason state legislatures mandate these 
tests is due to the perceived importance of 
standardized testing procedures that require 
all students to perform under similar circum- 
stances (Special Issues Analysis Center, 
1995). When using assessment portfolios, 
performance conditions may vary and 
teacher bias can affect students’ perfor- 
mance. For example, the variability in the 
amount of support teachers provide to 
students, the amount of time students are 
allowed to spend on portfolio samples, and 
the extent to which student work is aug- 
mented by support from external sources 
(e.g., peers, parents, tutors) has raised 
questions concerning the validity of 
inferences about student competence based 
on portfolio work (Gearhart, Herman, Baker, 
& Whittaker, 1993). 

Contrary to public perceptions, however, 
standardized testing conditions can be 
implemented for specific contents of assess- 
ment portfolios, such as a writing prompt. In 
Vermont, for example, the statewide writing 
prompt is administered to all students in 
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When; English language 
learners are included in a 

I 

portfolio system, the scoring 
criteria must be designed to 
assess content knowledge 
and language proficiency . 



grades 4 and 8 at the same time across the 
system. To ensure standardization, educa- 
tors at grades 4 and 8 receive staff develop- 
ment to learn which testing accommoda- 
tions are permissible and which are not. 

Cost 

One hurdle in developing an assessment 
portfolio system is the ongoing cost of the 
program. According to a RAND Corporation 
report on the cost of science performance 
assessments in large-scale testing programs, 
the cost of performance tests is three to ten 
times more expensive than multiple-choice 
tests (Stecher & Klein, 1997). However, 
science may be the most expensive content 
area for which to develop performance tasks 
because of the use of manipulatives (objects 
that students use to explore, apply, and 
connect different concepts and skills). 

Designing, implementing, and scoring 
portfolio items is labor intensive, and there- 
fore costly. Practitioners invest a substantial 
time commitment to align the assessment 
tasks with the curriculum, and to develop 
the scoring criteria and the scoring tools. 
Costs accrue throughout the development 
phase and into the implementation phase 
for both portfolio assessment and standard- 
ized tests. In fact, the only cost incurred by 
assessment portfolios that are not also 
incurred by standardized tests is that of , 
scoring the portfolio samples. 

State legislatures and school boards may 
inaccurately believe that standardized tests 
are less costly to design, administer, and 
score. When a school system purchases a 
contract with a commercial testing com- 
pany, and a one-time cost is incurred, this 
may be true. However, many states and 
districts now purchase tailor-made tests, the 
design and implementation of which are 
quite costly (O’Malley & Valdez Pierce, 
1994). Still, persuading state legislatures 



and school boards to agree to the additional 
costs involved in scoring an assessment 
portfolio system that includes English 
language learners can be difficult. 

Scoring 

Portfolios pose a difficult scoring dilemma 
for three reasons. First, developing and 
using scoring criteria requires labor-inten- 
sive staff discussion and training. When 
English language learners are included in a 
portfolio system, the scoring criteria must be 
designed to assess content knowledge and 
language proficiency. As the scoring criteria 
are developed, educators must keep second 
language acquisition principles in mind to 
ensure that the assessment criteria are valid 
and reliable for all students, including 
English language learners. Both content and 
form should be considered when applying 
the scoring criteria for assessment portfolios 
that include English language learners. To 
accommodate the needs of English language 
learners, additional weight can be given to 
content knowledge, as has been done in Vir- 
ginia's statewide writing test (O'Malley, M. 
personal communication, October 21, 

1997 ). 



School staff throughout the 
system agree on the outcomes 
being sought and the criteria 
being used to measure student 
performance, enabling 
stakeholders in a district to 
share a common goal. 



Second, developing inter-rater agreement 
among teachers and other scorers requires 
intensive staff development. The staff devel- 
opment consists of providing educators the 
opportunity to practice scoring student sam- 
ples in order to improve inter-rater agree- 
ment, as well as intensive staff discussion 
about the goals and objectives of the 
instructional program. Third, the process of 
scoring student work using predetermined 
scoring criteria is more time-consuming than 
the process of scoring a single-occasion, 
norm-referenced test. 

In Fairfax County, Virginia, a district- 
wide portfolio assessment system is used to 
assess and monitor the progress of all 
English language learners. ESL teachers in 
Fairfax have received extensive staff devel- 
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Conclusion 



D eveloping and implementing a 
large-scale assessment portfolio 
program that includes English language 
learners requires extensive planning and 
discussion and considerable resources. 
Many states and local districts have begun 
to use some performance assessments that 
require scoring with a rubric. A number of 
the perceived challenges of assessment 
portfolios (e.g., issues of reliability and 
comparability, ensuring standardized 
testing conditions, and the costs associated 
with developing, implementing, and scoring 
assessments based on content-area 
standards) also apply to other large-scale 
>rmance assessment measures that are 



scored with predetermined scoring criteria. 

To date, only Vermont and Kentucky 
use statewide portfolio assessment 
systems. Neither of these states have high 
proportions of English language learners; 
nor do they disaggregate the data for 
English language learners included in their 
assessment. District-wide assessment 
portfolios that include English language 
learners have begun to emerge but are not 
yet widespread. From these experiences, 
the advantages and challenges of 
assessment portfolios that include English 
language learners have been highlighted. 

The advantages are that stakeholders 
within the system have a common vision 
regarding what students should learn and 
be able to do, how goals will be assessed, 
and what criteria will be used. Improved 
teaching and learning are natural outcomes 



opment on the use and scoring of the port- 
folio samples. In most cases, teams of 
scorers, who have received intensive train- 
ing to improve inter-rater reliability figures, 
score writing samples based upon predeter- 
mined criteria. A subset of the portfolios is 
usually assessed by more than one scorer. 

Fortunately, as mentioned earlier, the 
staff development efforts needed to train 
teachers to use portfolios and to develop 
inter-rater reliability have an unexpected 
benefit. They result in teachers who share a 
common vision of what student 
achievement in the particular subject looks 
like. Teachers decide together what 
benchmarks to use for assessing student 
work. School staff throughout the system 
agree on the outcomes being sought and the 
criteria being used to measure student 
performance, enabling stakeholders in a 
district to share a common goal. Most 
norm-referenced and criterion-referenced 
tests cannot achieve this result. □ 



of a well-designed, well-implemented 
assessment portfolio system that includes 
professional development. Once a school 
system has decided to move forward with 
assessment portfolios that include English 
language learners, the lessons learned from 
the efforts of statewide portfolio systems, 
such as Vermont and Kentucky, and from 
emerging district-wide systems, such as 
San Diego and Fairfax County are 
invaluable. These voices from the field 
collectively recommend considering the 
steps described on the following page when 
designing a successful portfolio assessment 
system that includes English language 
learners. □ 






□ 



□ 



□ 



□ 

□ 



□ 

□ 



□ 



□ 

□ 



□ 




Steps to developing and implementing an assessment 
portfolio system that includes English language learners 

DECIDE ABOUT GOALS AND CONTENT: Stakeholders within the school and district, including those responsible 
for English language learners, decide what assessment information is needed and how that information can be 
provided. A group of teachers agree to lead the development of the portfolio program. 

DESIGN THE PORTFOLIO ASSESSMENT PROGRAM: The lead group of teachers, including ESL and bilingual 
education teachers, administrators, and parents decide on the range of products to be included in the portfolio 
assessment program. 

DEVELOP SCORING CRITERIA AND STANDARDS OF PERFORMANCE: The group decides upon common 
goals for student learning and performance and how they will be assessed, develops scoring rubrics and checklists, and 
agrees upon standards of performance to be attained. If possible, benchmarks that exemplify student work should be 
articulated, including benchmarks for English language learners. 

ALIGN TASKS TO STANDARDS AND CURRICULUM: The group aligns the assessment tasks to the district or 
state content standards and curriculum frameworks. 

IMPLEMENT AT PILOT SITES, PROVIDE STAFF DEVELOPMENT, AND ANALYZE RESULTS: Decide on 
pilot sites/classrooms and provide staff development on the implementation of portfolios. Following the 
implementation at pilot sites for at least one full school year, score the portfolios from the pilot sites. The effectiveness 
of the program should be assessed and the scoring criteria modified, based on the feedback of pilot site educators and 
the results of the scored portfolios. Study the effects of the portfolio assessment program on English language learners 
in particular, to determine whether improved information about these students is available as a result of portfolio 
implementation. 

IMPLEMENT AT ALL SITES: Once the program has been piloted and found to be effective, implement the portfolio 
program at all sites. 

TRAIN TEACHERS TO SCORE: School or district staff oversee the training of a team of teachers who practice 
scoring student work using the portfolio program’s scoring criteria and benchmarks. Training should include discussion 
of second language proficiency and its impact on student achievement. Efforts are made to reach an inter-rater 
reliability level of .7 - .8. 

ESTABLISH GUIDELINES FOR ADMINISTRATION: Stakeholders develop guidelines for a standardized 
collection of student work and decide the time, place, and manner in which standardized prompts will be given to 
assess students throughout the system. Accommodations for English language learners are delineated. 

. SCORE THE PORTFOLIOS: Teachers score the portfolios based upon predetermined criteria. This typically is done 
over several days in a central location by teachers who have been trained. 

REPORT THE RESULTS: All stakeholders receive information about the results of the portfolio assessment in a 
timely fashion, in ways that make the results meaningful to all, including teachers, students, parents, and other 
community members. 

EVALUATE THE PROGRAM: After one year, evaluate the effectiveness of the portfolio program and make necessary 
judgments. 
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